Working Together: development of synthetic actors, alongside human actors, in rehearsals

While film has long utilised computers to create otherwise impossible characters, similar use within live stage performances is relatively uncommon. Even within film, these characters have, until recently, been mere puppets controlled by some form of puppeteer. This paper outlines research carried out at Massey University into autonomous synthetic (computer) actors that are capable of learning through the rehearsal process, alongside human counterparts. In particular, the paper describes a production of Orson Scott Card’s Porcelain Salamander, which utilised synthetic actors for one main role, and various minor roles, as well as four human actors. Throughout the rehearsal process, the synthetic characters refined their performance, through the typical loop of improvisation followed by feedback from a director, until their final performance was optimal. The synthetic actors utilised visual data (from cameras), audio data (from a central microphone), and were actuated via projected graphical displays and output into the theatre sound system. This paper lightly covers the technical aspects of the learning and improvisational techniques utilised by the synthetic actors. It concentrates more on phenomenological analysis of qualitative data obtained through the rehearsal process, and quantitative analysis of audience reaction to the final performance. These results show that synthetic actors are capable of rapid learning with the very limited testing and training time available in rehearsals. The results also demonstrate the ability for the synthetic actors to contribute, in a limited fashion, to the development of the performance, as human actors do. 1. A History of the Use of Computers in Theatrical Artistic Processes The film and music industries have been extensively using computers in artistic processes for decades [22], while theatrical productions have rarely done so. Pinhanez provides a lengthy definition of the term ‘computer theatre’: “[it] is about providing means to enhance the artistic possibilities and experiences of professional and amateur actors, or of audiences clearly engaged in a representational role in a performance” [19]. This definition “restrict[s] the usage of the term computer theatre to performance situations (ruling out, for instance, user browsing and story-telling)” [19]. In brief, we are concerned with live, real-time, theatrical performances which use computers in the artistic process. 1.1. Scenery One use for computers in a live stage performance is generation of the scenery [24]; this is a logical progression from the use of lighting to set a scene, and also allows more rapid and dynamic scenery, well suited to generations that have grown up in front on the television, or watching movies, rather than at the theatre [31]. A group at the University of Kansas began testing virtual reality as a tool for traditional scenography [30], and then moved on to using virtual reality in productions with the production of The Adding Machine in 1996 [27], Samuel Beckett’s Play [25], an adaptation of Arthur Kopit’s Wings [26], David Gustav Fraser’s Tesla Electric in 1998 [28], Machinal in 1999 [29], A Midsummer Night’s Dream in 2000 [32] and Dinosaurus in 2001 [33]. Just as computer generated images are integrated into live action sequences in film, Monsters of Grace, from the Brooklyn Academy of Music in 1998 [11], integrated computer generated scenes generated prior to the performance into an opera. The music for the opera was conducted and performed live for each performance, but the imagery (projected onto a screen) was developed prior to the performances. 1.2. Remote collaboration The simplest form of remote collaboration is utilising computer technology to allow a number of authors to jointly develop a script for a performance, as in Stations of Desire [14] and the work of the Plaintext Players [23]. More complex collaboration occurs during real-time, for example the ‘Desktop Theatre’ [35] performance of Beckett’s Waiting for Godot, Santaman’s Harvest, Spectacled Society, World of Park, and Water Wars[2]. These performances (other than Water Wars[2]) did not have a live component – audiences watched it on any desktop computer with an Internet connection via chat software. Avatar Body Collision use similar technology, but incorporate a live performer into the performance, as in Screen Save Her, the[abc]experiment [13], s w i m, and BodyBeat. 1.3. Synthetic actors In 1995, Barbara Hayes-Roth suggested that artificial intelligence researchers expand their purview to include characters and actors whose performances are designed to create particular kinds of impressions on their human audiences [12]. In Play, one of their experiments in virtual scenography, the University of Kansas group utilised a “piercing spotlight” as a “live, visible agent”, although the spotlight was still under direct human control. Synthetic actors are similar to digital characters in film, but are (to varying extents) autonomous. These characters range from those that only interact with other synthetic characters, to those that interact with humans as well; from characters that interact with a single user via a computer to characters that entertain a large audience in a traditional theatre. From 1996 to 2001 Claudio Pinhanez at the Media Lab at M.I.T. produced various experiments in computer theatre, concentrating on computer, or synthetic, actors. In 1996’s SingSong, four disembodied heads interacted with a human conductor to perform a short musical item in front of a live audience [20]. All the characters were completely autonomous, working off a known script, and able to adapt to (limited) changes in the performance by the human actor. 1998’s It/I told the story of a computer actor (It) who tries to keep the attention of a human actor (I) by presenting various images, generating sounds, and changing lighting. It is an example of a disembodied computer actor, similar to the spotlight in Play [22]. It/I was followed by It, an interactive exhibit, based on It/I [20]. Pinhanez’s actors were able to adapt to temporal changes in the performance, and, to a certain extent, vary their performance each time it was presented [21]. The Virtual Theater project at Stanford University aimed to provide a multimedia environment in which users could fill all the roles associated with putting on a play in an improvisational theatre company, including producer, playwright, casting directory, set designer, music director, real-time directory, and actor. Those roles not filled by the user were filled by intelligent agents. Productions that formed a part of the Virtual Theater project included the Woggles, CyberCafe, The Forest Sauvage, Tigrito, and Master/Servant Scenarios. The focus of the Virtual Theater project was on improvisation in the classical mode of improvisation first practiced by the Commedia dell’Arte during the Italian Renaissance, rather than performances that were scripted [36]. Ideally, synthetic actors are capable of following a script, but also reacting according to its own role. Reilly and Bates state, “We are not necessarily interested in lifelike or realistic agents. We don't really care if our characters only have four fingers or are talking animals. The arts have always abstracted away from reality and created characters that were ‘larger than life’ and that were much more interesting than any mere ‘real’ people or animals we might know. We [want] interactive versions of these abstracted characters that have been so successful in non-interactive media” [34] Some synthetic characters are limited to interacting with users via the traditional human-computer interface, rather than in a theatrical setting. Mateas’s group aims to develop interactive drama that builds “dramatically interesting virtual worlds inhabited by computer-controlled characters, within which the user ... experiences a story from a first person perspective” [15]. A key point is that “it should not just be an interesting demo in a closed door lab, but be experienced by people in the real world. Ultimately, this is the only way to validate the ideas.” [15]. Synthetic actors are also, of course, used outside the theatrical domain – for example in commentating soccer games [4], agent research [6], and education [18]. Computers have also been used in other ‘creative’ areas, such as generating music [2; 3; 5] and line drawings [5]. 2. The Porcelain Salamander 2.1. Story Orson Scott Card created the short story The Porcelain Salamander [8] as a bedtime story for his wife, later publishing it in Unaccompanied Sonata and Other Stories [9] and then in Cardography [10]. Although it is not among his most well known stories, it, one of the briefest he has written, is often judged to be among his best [9]. The fable describes a young girl, Kiren, whose mother dies in childbirth; her father is so distraught at this that he curses Kiren to never move a muscle in her life, until she loses someone she loves as much as he loved her mother. Although the curse isn’t that strong, Kiren grows up barely capable to move, while her father, who regretted the curse immediately afterwards, spends his time searching for a cure. At the start of the story, Kiren’s father meets a stranger on his journey who claims to have a cure for Kiren, and gives her father a porcelain salamander to give to Kiren. The salamander moves, as if by magic, and can speak (although this is only known to Kiren), but must never stop moving, or will transform to ordinary lifeless porcelain. Kiren grows to love the salamander – to the point where she loves it as much as her father loved her mother. The stranger then arrives at Kiren’s house, and causes Kiren and the salamander to be trapped behind a wall, which gradually creeps in on them. In Kiren’s weak state, she cannot jump out, although she could climb out if she had something to stand on. The salamander decides to sacrifice himself, so that Kiren can stand on him and climb out. Through the salamander’s sacrifice Kiren is able to live out the rest of her life unburdened by the curse. Although she is initially devastated and angry at the loss of the salamander, she eventually comes to realise that the salamander was finally able to rest, frozen forever in a moment of perfection that rarely comes to people, and, in true ‘happy ever after’ style, as she grows up she gains fame and beauty. 2.2. Story Choice and Parallels The Porcelain Salamander is not a story that could be conventionally told via a stage performance without changing the salamander character to something more humansized. As such, one of the primary reasons for selecting the story was that it could make use of the technology without being about the technology. The salamander also shared many characteristics of a synthetic actor. It was not truly alive, but had the appearance of life; it did not feel emotions, but was able to draw out emotional responses from those that it interacted with. It can move and speak, but not naturally, and if it ever stopped moving (processing), it would die (cease to operate). 2.3. Technical Setup The Porcelain Salamander was performed at Massey University’s Underground Theatre as a short (25 minute) demonstration of the use of synthetic actors in live stage theatre. The cast included four human actors and four synthetic characters (the main role of the salamander, and very brief appearances by a spider, a bird, and a moving wall). Rehearsals ran from March to July 2004, with performances held on the 30 and 31 of July 2004. The performance included one moderately sized set with the front of Kiren’s house on stage left and the woods near her house centre stage and stage right (see Figure 1). A large (approximately 4m wide, 6m high) front-projected screen was upstage centre (see Figure 2) and a smaller (approximately 2m wide, 1.5m high) rear-projected screen formed the back of the woods on stage right (see Figure 3). Five networked computers handled the run-time synthetic actor system; one located hidden onstage, three located offstage behind the stage right woods, and one in another campus Figure 1: The Porcelain Salamander Set building. During some of the rehearsals up to four nodes of a small cluster located in another campus building were also utilised for the run-time system. The run-time synthetic actor system included five main components for The Porcelain Salamander: speech recognition, speech synthesis, graphical representation, a simple vision system, and the control system. The speech recognition and synthesis (powered by MicrosoftTM’s SAPI speech system) ran on one computer, the vision system on another, the control system on a third, and one instance of the graphics (see Figures 4,5) system (powered by Unreal TournamentTM 2003) on each of two others (each screen required a separate computer for generation of the graphics). 2.4. Rehearsals Figure 2: Human and synthetic actors on stage 16 rehearsals were held between March 15 and July 3